Mobile terminal device, operation method, program, and storage medium

ABSTRACT

A mobile terminal device includes a touch panel where a specified input operation is performed with an indicator, an input detection unit, and an operation determination unit. The input detection unit detects the pattern of the input operation based on the position of the indicator on the touch panel, and decreases the frequency of execution of processing arranged to detect the input operation performed on the touch panel when an input operation performed on the touch panel is not detected over a specified time period. When the pattern of the input operation is detected in the case where the execution frequency of the processing arranged to detect the input operation is low, the operation determination unit causes a controlled unit determined to be an operation target in relation to the pattern of the input operation to execute a specified operation.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.15/833,470, filed Dec. 6, 2017, which is a continuation of U.S. patentapplication Ser. No. 15/398,080, filed Jan. 4, 2017 (now U.S. Pat. No.9,864,512), which is a continuation of Ser. No. 15/159,238, filed May19, 2016 (now U.S. Pat. No. 9,569,096), which is a continuation of U.S.patent application Ser. No. 14/800,229, filed Jul. 15, 2015 (now U.S.Pat. No. 9,507,401), which is a continuation of U.S. patent applicationSer. No. 13/926,744, filed Jun. 25, 2013 (now U.S. Pat. No. 9,116,696),which is based upon and claims the benefit of the earlier filing date ofU.S. Provisional Patent Application No. 61/672,953 filed in the USPTO onJul. 18, 2012; the entire contents of which being incorporated herein byreference in its entirety.

TECHNICAL FIELD

The present disclosure relates to a mobile terminal device, an operationmethod, a program, and a storage medium that are applied to the casewhere a user performs an input operation for a touch panel to execute aspecific function.

BACKGROUND ART

In recent years, touch panels have been increasingly adopted forterminal devices including a mobile phone terminal, a highperformance-type mobile information terminal having a conversationfunction, what is referred to as a smart phone, a tablet-type terminal,and so forth, as input devices for which users perform an operationinput. The touch panel is a device including a display unit displayingan image, etc. and a position input device such as a touch sensor, whichare integrated into one body.

In a terminal device including a touch panel, specified operations areinput by a user to various icons that are displayed on the screen of thedisplay unit, for example, so that application software (hereinafteroften abbreviated to “application”) executes. The application isassociated with the icon, and performs an operation in accordance withthe details of the operation input. As operation input methods, a tapoperation achieved by tapping the screen, a drag operation achieved bymoving an indicator including a touched finger, a stylus pen, etc. froma specified position to a different position, a flick operation achievedby moving the finger or the indicator on the screen in such a way tomake a quick sweeping stroke, a tap operation achieved by tapping thescreen, and so forth have been available.

The various operations are determined based on, for example, informationabout a coordinate position defined on the screen where a finger of theuser or the indicator touches or approaches, which is attained during aseries of operations performed from when the finger or the indicatortouches the screen to when the finger or the indicator leaves thescreen. Specifically, information about, for example, the amount of achange in the coordinate position per unit time, the moving speed of thecoordinate position, an area on the screen where a touch is detected,etc. is used as information specifying the details of an operationinput.

Then, in the mobile terminal device, a shift to a suspend state which isa power saving mode is made when there is no operation input from theuser over a fixed time period, so as to reduce the power consumption. Inthe suspend state, the light amount of backlight of the display unit isreduced, or various functions are stopped to reduce the powerconsumption. However, since the details of an operation input performedbefore making the shift to the suspend state, etc. are stored in amemory, processing can be started again by reading the details of theoperation input, etc. from the memory after waking up from the suspendstate (referred to as “wakeup”). In the following description,processing performed to wake up from the suspend state is referred to as“resuming”.

FIGS. 17A and 17B are schematic diagrams illustrating an exemplaryexternal configuration of a known mobile terminal device 100. FIG. 17Aillustrates an exemplary operation performed to wake up the mobileterminal device 100 from the suspend state, and FIG. 17B illustrates anexample where an application is started through resuming and icons, etc.are displayed on a display unit 102.

The mobile terminal device 100 includes a touch panel 101, the displayunit 102 superimposed on the touch panel 101, and a home button 103provided to display a home screen image. When the mobile terminal device100 is in the suspend state, a user performs an operation by pressingthe home button 103 with a finger (see FIG. 17A). When the mobileterminal device 100 wakes up through resuming, the home screen image isdisplayed on the display unit 102 (see FIG. 17B). On the home screenimage, lists of various types of video, functional icons are displayed.

Further, the method of starting the operation of a device by detectingan operation performed on the touch panel has also become available. Forexample, in Patent Literature 1, the method of shifting a device to auser-interface unlocked state when a contact with a touch sensitivedisplay corresponds to a specified gesture is disclosed.

CITATION LIST Patent Literature

[PTL 1] Description of U.S. Pat. No. 8,046,721

SUMMARY OF INVENTION

As stated above, in the known mobile terminal device 100, each unit isresumed after the user presses the home button 103, or after a specifiedgesture operation is input by the user. Then, an operation includingselecting the functional icon corresponding to an application to bestarted anew after the home screen image is displayed and starting theapplication is performed. However, it is much trouble to perform theoperation including selecting the functional icon after going throughthe home screen image each time resuming is performed, which makes itimpossible to use a target function immediately. Further, when resumingis performed, initialization processing, etc. are needed for each targetfunction, which becomes a cause of retarding the startup of anapplication.

Accordingly, the inventor recognizes the necessity to immediatelyexecute a specific function when in the suspend state.

A mobile terminal device according to an embodiment of the presentdisclosure includes a touch panel where a specified input operation isperformed with an indicator, an input detection unit configured todetect a pattern of the input operation based on a position of theindicator on the touch panel, and decrease a frequency of execution ofprocessing arranged to detect the input operation performed on the touchpanel when the input operation performed on the touch panel is notdetected over a specified time period, and an operation determinationunit configured to instruct a controlled unit determined to be anoperation target based on the pattern of the input operation to executea specified operation when the pattern of the input operation isdetected in the case where the execution frequency of the processingarranged to detect the input operation is low.

Further, an operation method according to an embodiment of the presentdisclosure includes a step of detecting a pattern of an input operationbased on a position of an indicator by which an input operation isperformed on a touch panel, a step of detecting the pattern of the inputoperation based on the position of the indicator on the touch panel, anddecreasing a frequency of execution of processing arranged to detect theinput operation performed on the touch panel when the input operationperformed on the touch panel is not detected over a specified timeperiod, and a step of instructing a controlled unit determined to be anoperation target based on the pattern of the input operation to executea specified operation when the pattern of the input operation isdetected in the case where the execution frequency of the processingarranged to detect the input operation is low.

Further, a program according to an embodiment of the present disclosurecauses a computer to execute a procedure for detecting a pattern of aninput operation based on a position of an indicator by which an inputoperation is performed on a touch panel, a procedure for detecting thepattern of the input operation based on the position of the indicator onthe touch panel, and decreasing a frequency of execution of processingarranged to detect the input operation performed on the touch panel whenthe input operation performed on the touch panel is not detected over aspecified time period, and a procedure for instructing a controlled unitdetermined to be an operation target based on the pattern of the inputoperation to execute a specified operation when the pattern of the inputoperation is detected in the case where the execution frequency of theprocessing arranged to detect the input operation is low.

Further, a computer-readable storage medium according to an embodimentof the present disclosure stores a program causing a computer to executea procedure for detecting a pattern of an input operation based on aposition of an indicator by which an input operation is performed on atouch panel, a procedure for detecting the pattern of the inputoperation based on the position of the indicator on the touch panel, anddecreasing a frequency of execution of processing arranged to detect theinput operation performed on the touch panel when the input operationperformed on the touch panel is not detected over a specified timeperiod, and a procedure for instructing a controlled unit determined tobe an operation target based on the pattern of the input operation toexecute a specified operation when the pattern of the input operation isdetected in the case where the execution frequency of the processingarranged to detect the input operation is low.

According to the present disclosure, when an input operation performedon the touch panel is not detected over a specified time period, thepower consumption is reduced by decreasing the execution frequency ofprocessing arranged to detect the input operation performed on the touchpanel. Then, when the pattern of the input operation is detected whenthe execution frequency of the processing is decreased, a controlledunit determined to be an operation target in relation to the pattern ofthe input operation is caused to execute a specified operation.Accordingly, a specific function can be immediately executed when noinput operation is performed and a shift to the suspend state is made.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram illustrating an exemplary externalconfiguration of a mobile terminal device according to a firstembodiment of the present disclosure.

FIG. 2 is a block diagram illustrating an exemplary internalconfiguration of the mobile terminal device according to the firstembodiment of the present disclosure.

FIG. 3 is an explanatory diagram illustrating an exemplary layerconfiguration of software and hardware that operate in the mobileterminal device according to the first embodiment of the presentdisclosure.

FIGS. 4A and 4B are explanatory diagrams illustrating examples of ausual scan and an intermittent scan that are performed according to thefirst embodiment of the present disclosure.

FIGS. 5A and 5B are explanatory diagrams illustrating exemplarycapacitance values that are detected with a touch sensor on an operationsurface during a touch operation performed in a usual state according tothe first embodiment of the present disclosure.

FIGS. 6A and 6B are explanatory diagrams illustrating exemplarycapacitance values that are detected with the touch sensor on theoperation surface in the state where the mobile terminal deviceaccording to the first embodiment of the present disclosure is in apocket, etc. of a user.

FIGS. 7A and 7B are explanatory diagrams illustrating an example where ahome screen image is displayed from a suspend state through a firstgesture operation performed according to the first embodiment of thepresent disclosure.

FIGS. 8A and 8B are explanatory diagrams illustrating an example wherean imaging function is started from the suspend state and a picture isshot through a second gesture operation performed according to the firstembodiment of the present disclosure.

FIG. 9 is a sequence diagram illustrating exemplary processing executedto start a camera application and perform shooting through a knownoperation.

FIG. 10 is a sequence diagram illustrating exemplary processing executedto start the camera application and perform shooting through anoperation performed according to the first embodiment of the presentdisclosure.

FIGS. 11A and 11B are explanatory diagrams illustrating an example wherea music reproduction function is started from the suspend state and amusic file is reproduced through a third gesture operation performedaccording to the first embodiment of the present disclosure.

FIG. 12 is a sequence diagram illustrating exemplary processing executedto start a music reproduction application 23 and reproduce music throughan operation performed according to the first embodiment of the presentdisclosure.

FIG. 13 is a flowchart illustrating exemplary processing performed toexecute various functions by combining a gesture operation performedaccording to a second embodiment of the present disclosure with anacceleration value detected with an acceleration sensor.

FIGS. 14A, 14B, 14C, 14D and 14E are explanatory diagrams illustratingan exemplary operation arranged to start an application through atwo-step gesture operation performed according to a third embodiment ofthe present disclosure.

FIGS. 15A, 15B, 15C, and 15D are explanatory diagrams illustratingexemplary assistance processing performed when a gesture operation endsin failure according to a fourth embodiment of the present disclosure.

FIG. 16 is a sequence diagram illustrating exemplary processingperformed to execute an assistance function according to the fourthembodiment of the present disclosure.

FIGS. 17A and 17B are schematic diagrams illustrating an exemplaryexternal configuration of a known mobile terminal device.

FIG. 18 is a schematic diagram illustrating an exemplary externalconfiguration of a mobile terminal device according to a fifthembodiment of the present disclosure.

FIGS. 19A and 19B are explanatory diagrams illustrating an example wherea mobile terminal device is held by one hand on the lateral sides andexemplary object detected area of side panels according to the fifthembodiment of the present disclosure.

FIG. 20 is a sequence diagram illustrating an exemplary processingperformed to wake up a mobile terminal device by detecting holding ofthe mobile terminal device by side panels according to the fifthembodiment of the present disclosure.

FIGS. 21A and 21B are explanatory diagrams illustrating an example wherea mobile terminal device is held to capture (or shoot) an image by twohands on the lateral sides and exemplary object detected area of sidepanels according to the fifth embodiment of the present disclosure.

FIG. 22 is a sequence diagram illustrating an exemplary processingexecuted to start the camera application and perform shooting through anoperation detected by side panels according to the fifth embodiment ofthe present disclosure.

FIGS. 23A and 23B are explanatory diagrams illustrating an example wherea mobile terminal device is held by one hand avoiding touching sidepanels and exemplary object detected area of side panels according tothe fifth embodiment of the present disclosure.

FIGS. 24A and 24B are explanatory diagrams illustrating an example wherea mobile terminal device is held by one hand and exemplary objectdetected area of a front panel according to the fifth embodiment of thepresent disclosure.

FIG. 25 is a sequence diagram illustrating an exemplary processingperformed to wake up a mobile terminal device by detecting holding ofthe mobile terminal device by a front panel according to the fifthembodiment of the present disclosure.

FIGS. 26A and 26B are explanatory diagrams illustrating an example wherea mobile terminal device is held to shoot by two hands and exemplaryobject detected area of front panel according to the fifth embodiment ofthe present disclosure.

FIG. 27 is a sequence diagram illustrating an exemplary processingexecuted to start the camera application and perform shooting through anoperation detected by a front panel according to the fifth embodiment ofthe present disclosure.

FIGS. 28A and 28B are explanatory diagrams illustrating an example wherea mobile terminal device is held by one hand avoiding touching a frontpanel and exemplary object detected area of a front panel according tothe fifth embodiment of the present disclosure.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present disclosure will be described.Incidentally, the descriptions will be given in the following order.

<First Embodiment>

1. Exemplary External configuration of Mobile terminal device2. Exemplary Internal configuration of Mobile terminal device3. Exemplary Software configuration4. Exemplary Scans performed for Touch sensor5. Exemplary Gesture operations

5-1. Exemplary Gesture operations performed to start Home screen image

5-2. Exemplary Gesture operation performed to execute Shooting

5-3. Exemplary Gesture operation executed to perform Music reproduction

<Second Embodiment>

6. Example where Operation is performed in concert with Another device

<Third Embodiment>

7. Example where Application is started through Two-step gestureoperation

<Fourth Embodiment>

8. Exemplary Assistance function started when Gesture operation ends infailure

Hereinafter, a first embodiment of the present disclosure will bedescribed with reference to FIG. 1 to FIG. 12. In the presentembodiment, an exemplary application of a mobile terminal device 10configured to immediately start a specific function in response to anoperation performed for a touch panel 7 will be described. The mobileterminal device 10 achieves an operation method performed by internalblocks working in cooperation with one another, which will be describedlater, through a computer executing a program. Incidentally, commoncomponents that are shown in the drawings are designated by the samereference signs.

First Embodiment [1. Exemplary External Configuration of Mobile TerminalDevice]

FIG. 1 is a schematic diagram illustrating an exemplary externalconfiguration of the mobile terminal device 10.

The mobile terminal device 10 includes a cabinet 19, a touch panel 7where a specified input operation is performed with an indicator, thetouch panel 7 being accommodated in the front part of the cabinet 19,and operation keys 8 that are provided on, for example, the end part ofthe front of the cabinet 19.

The touch panel 7 is employed as an input device for which a userperforms an input operation for a terminal device including a mobilephone terminal, a high performance-type mobile information terminalhaving a conversation function, what is referred to as a smart phone, atablet-type terminal, and so forth. The touch panel 7 is configured as,for example, a sensor achieved under the capacitance system. Then, thetouch panel 7 includes a touch sensor 7 (see FIG. 2) used as a positioninput device to which a position specified by an indicator including afinger, a stylus pen, etc. is input and a display unit 12 displaying animage, etc., which are integrated into one body. The display unit 12includes, for example, a liquid crystal panel or an organic EL (ElectroLuminescence) panel, etc.

As a sensing system allowing the touch panel 7 to detect an inputoperation achieved by the indicator, the following systems may be used.For example, the resistive film system detecting a change in theresistance value of a part touched by the indicator, the electromagneticinduction system detecting the electromagnetic energy of an indicatorspecifically designed to generate a magnetic field, the capacitancesystem detecting a change in the capacitance value of a touched part,etc. may be used.

Hereinafter, an example where a finger is used as the indicatorperforming an input operation on the touch panel 7 will be described.

As for the terminal device 10, the user inputs specified operations forvarious icons that are displayed on the screen of the display unit 12,for example, so that various applications that are associated with theicons perform operations based on the details of the operation inputs.The details of the input operations include a tap operation achieved bytapping the surface of the touch panel 7, a drag operation achieved bymoving a finger caused to touch the touch panel 7 from a specifiedposition to a different position, a flick operation achieved by movingthe finger on the screen in such a way to make a quick sweeping stroke,and so forth.

The various operations are determined based on, for example, informationabout coordinates on the screen where a finger touches or approachesduring a series of operations performed from when the finger of the useror a finger touches the screen to when the finger of the user or thefinger leaves the screen. Specifically, information about, for example,the amount of a change in the coordinates per unit time, the movingspeed of the coordinates, an area on the screen where a touch isdetected, etc. is used as information specifying the details of anoperation input.

The operation keys 8 include a home button, a power button, a volumebutton, etc. The user can display a home screen image on the displayunit 12 by pressing the home button. Further, the power of the mobileterminal device 10 can be turned on/off by pressing the power button.Further, the magnitude of volume can be changed during a conversation orduring the music reproduction by pressing the volume button.

[2. Exemplary Internal Configuration of Mobile Terminal Device]

FIG. 2 is a block diagram illustrating an exemplary internalconfiguration of the mobile terminal device 10.

The mobile terminal device 10 includes an antenna 1 performingtransmission and reception of an electric wave between a radio telephonebase station, a communication processing unit 2 to which the antenna 1is connected, and a control unit 3. Further, the mobile terminal device10 includes a voice processing unit 4, a speaker 5, and a microphone 6.

The communication processing unit 2 performs radio communications withthe radio telephone base station under control of the control unit 3.During a voice conversation, voice data included in data received withthe communication processing unit 2 is supplied to the voice processingunit 4 that performs voice processing for the voice data and that causesthe speaker 5 to emit a voice. Then, the voice processing unit 4performs decoding processing for the voice data so that an analog voicesignal is obtained. The analog voice signal obtained with the voiceprocessing unit 4 is supplied to the speaker 5, which is an exemplarysound emitting unit, and output as a voice. A voice signal collected andacquired with the microphone 6 is also supplied to the voice processingunit 4, and encoded into voice data in a specified encoding format(e.g., PWM (Pulse Width Modulation)) with the voice processing unit 4.Then, the acquired voice data is supplied to the communicationprocessing unit 2, and emitted through the antenna 1 as a radio wave.

The processing units including the communication processing unit 2, thevoice processing unit 4, and so forth exchange control data with thecontrol unit 3 via a control line CL, and transfer data via a data lineDL. The control unit 3 includes a CPU (Central Processing Unit), a GPU(Graphics Processing Unit), and so forth, and controls the unitsconstituting the mobile terminal device 10. For example, the controlunit 3 performs changing a command issued for application softwarestored in a storage unit 3 that will be described later, etc. based on aresult of the operation determination performed with a touch sensordrive unit 72 that will be described later.

Further, the mobile terminal device 10 includes the touch panel 7 andthe operation keys 8. In the touch panel 7, the touch sensor 71, thetouch sensor drive unit 72, and the display unit 12 are included. Thetouch sensor 71 is configured as, for example, a sensor achieved underthe mutual capacitance system of the projected capacitance system, andincludes transmission electrodes (TX) and reception electrodes (RX) thatare not shown, which are arranged in matrix form in the X direction andthe Y direction.

By grounding the reception electrode and inputting a drive pulse to thetransmission electrode, an electric field is generated between thetransmission electrode and the reception electrode. When an indicatorincluding a finger of the user, which is a kind of electric conductor, astylus pen, etc., approaches the surface of the touch panel 7(hereinafter referred to as the “operation surface”), part of theelectric field is absorbed in the electric conductor so that theelectric field generated between the transmission electrode and thereception electrode is decreased. Consequently, an electric charge(capacitance value) occurring in the reception electrode is alsodecreased. Accordingly, the measurement of a decrease in the electriccharge, the decrease being associated with a decrease in the electricfield generated between the transmission electrode and the receptionelectrode, allows for detecting that the operation surface of the touchpanel 7 is touched/untouched.

The touch sensor drive unit 72 inputs a drive pulse to each transmissionelectrode, and detects an electric charge of the reception electrode.The touch sensor 71 is used as an input detection unit that detectswhether the indicator for which an input operation is performed touchesor untouches the operation surface of the touch panel 7 by measuring achange in the capacitance value detected with the touch sensor driveunit 72.

The touch sensor 71 detects a position defined on the operation surfaceof the touch panel 7 where a touch or an approach of the indicator isdetected, and detects the position of the indicator on the touch panel7. Then, the touch sensor 71 also outputs information about the XYcoordinates corresponding to that position. As for the details of aninput operation performed by the user, a pinch (pinch-in/pinch-out)operation achieved by decreasing and increasing the space between twofingers, etc. are performed in addition to the above-described tapoperation, drag operation, and flick operation. Further, the touchsensor 71 detects the pattern of an input operation performed by theindicator on the touch panel 7 based on a position (coordinateinformation, etc.) defined on the screen where the indicator touches orapproaches during a series of operations performed from when theindicator touches on the operation surface to when the indicator leavesthe operation surface.

The touch sensor drive unit 72 is used as an operation determinationunit that determines the type of the indicator (whether or not it is afinger, etc.) that touches or approaches the operation surface of thetouch panel 7 based on a capacitance value detected with the touchsensor 71. The capacitance value includes, for example, informationabout a distribution attained on the operation surface of the touchpanel 7, the magnitude of the capacitance value, and so forth.

The control unit 3 controls the display processing unit 9 to performcontrol to cause the display unit 12 to display a specified screenimage.

The display processing unit 9 causes the display unit 12 to display thescreen image corresponding to an application executed with the controlunit 3. For example, the display unit 12 is caused to display videotaken by a camera application. Further, a browser or the like is oftenscrolled up or down for display based on the details of an operationperformed on the operation surface of the touch panel 7, the screenimage is often moved based on the flick operation, and a selected spotis often reduced or enlarged for display based on the pinch operation.For displaying an image on the display unit 12, the display processingunit 9 stores several frames' worth of video data in a frame buffer 11.The frame buffer 11 is a memory storing video data on FIFO (First In,First Out) basis. The display unit 12 displays video based on the videodata of each frame read from the frame buffer 11.

Further, the mobile terminal device 10 includes an imaging drive unit13, an imaging unit 14, a storage unit 15, an antenna 16, a radiocommunication processing unit 17, and a sensor unit 18.

The imaging unit 14 includes an imaging element 14 a that has an imagesensor including, for example, a CCD (Charge Coupled Device), a CMOS(Complementary Metal Oxide Semiconductor), and so forth. Further, theimaging unit 14 has an optical system including an un-illustrated lensprovided to photograph a subject, a shutter, and so forth. Then, theimaging unit 14 generates an image signal by performing photoelectricconversion for a subject light formed on the light receiving surfacethereof via the un-illustrated lens. An imaging operation of the imagingunit 14 is controlled with the imaging drive unit 13.

The voice processing unit 4 and the speaker 5, the display processingunit 9, the frame buffer 11 and the display unit 12, and the imagingdrive unit 13 and the imaging unit 14 are used as controlled units. Theoperations of the controlled units are controlled via middleware when aninput operation is performed on the touch panel 7 in the state whereapplication software is started, and the operations are controlled viathe middleware in a suspend state.

Then, when the pattern of the input operation is detected when thefrequency of execution of processing arranged to detect an inputoperation is low, the touch sensor drive unit 72 instructs a controlledunit which is determined to be an operation target based on the patternof the input operation to execute a specified operation. Further, whenthe pattern of the input operation is detected in the suspend statewhere at least the application software and the middleware are notstarted, the touch sensor drive unit 72 starts the middleware that willbe described later. Then, a specified operation is executed from themiddleware to the controlled unit which is determined to be theoperation target in relation to the pattern of the input operation.

The storage unit 15 includes a ROM (Read Only Memory) or a RAM (RandomAccess Memory), and stores various types of application software,necessary data for executing those, and so forth. The radiocommunication processing unit 17 communicates with peripheral devices byradio based on procedures conforming to a specified communicationstandard via the connected antenna 16.

The sensor unit 18 includes an acceleration sensor, a magnetic fieldsensor, etc., and outputs a specified sensor value. Due to the sensoroutput of the acceleration sensor, the control unit 3 can determine anoperation achieved by lifting and shaking the mobile terminal device 10,and the orientation in which the mobile terminal device 10 is used.

[3. Exemplary Layer Configuration]

FIG. 3 is an explanatory diagram illustrating an exemplary layerconfiguration of software and hardware that operate in the mobileterminal device 10.

In the mobile terminal device 10, an HMI (Human Machine Interface)layer, a middleware layer, and a hardware layer are set, and variousmodules operate in relation to one another in each layer. Module groupsthat are included in the HMI layer and the middle layer are referred toas the software in contrast to the hardware.

In the following description, software administering the operations ofapplications and middleware that are included in each layer, and theunits that are included in the mobile terminal device 10, and theenergization, etc. of the units under control of the control unit 3 isreferred to as a “system”. The system includes, for example, an OS(Operating System), etc. The system performs processing includinginterrupt control, timeout management, etc., outputs variousinstructions, etc., collectively manages the operations of the mobileterminal device 10, and performs control to execute resume processingand suspend processing in each unit, for example.

In the hardware layer, the units that are provided in the mobileterminal device 10, which are illustrated in FIG. 2, are included.Between the HMI layer and the middleware layer, data is mutuallytransferred via a public API (Application Programming Interface) thatmediates the processing of various types of application software.Between the middleware layer and the hardware layer, data is mutuallytransferred via a driver API mediating processing performed for theunits that are included in the mobile terminal device 10.

In the HMI layer, various modules including a home screen imageapplication 21, a camera application 22, a music reproductionapplication 23, etc. are provided. The home screen image application 21performs control to display a home screen image on the display unit 12.The camera application 22 performs control to display various menus,etc. that are provided to perform imaging through the imaging unit 14.The music reproduction application 23 performs control to cause thevoice processing unit 4 to perform codec of voice data, for example.

In the middleware layer, a quick start 31, camera middleware 32, andmultimedia middleware 33 are provided. The quick start 31 performscontrol to resume the system of the mobile terminal device 10 which isin the suspend state. Further, the quick start 31 is used as aninterface which starts each of the units of the hardware layer with highspeed by performing sequence control during resume processing. As afunction allowing the quick start 31 to perform the sequence control,there is one that unarchives a music file of which reproduction isinterrupted in a memory and that increases the speed of a read operationto control a series of operations necessary to execute an imagingfunction and execute a music reproduction function, for example.

The camera middleware 32 converts and transfers an instruction issuedfrom the camera application 22 to the imaging drive unit 13, andnotifies the camera application 22 of the operation state of the imagingunit 14, etc., which is received from the imaging drive unit 13. Themultimedia middleware 33 accumulates video data, etc. in the framebuffer 11, and outputs the video data read from the frame buffer 11 tothe display unit 12 to display video. Further, the multimedia middleware33 causes the voice processing unit 4 to perform codec processing, etc.for voice data, etc., and outputs the processed voice data to thespeaker 5 to emit a voice.

Incidentally, in the past, during processing performed to wake up thesystem which is in the suspend state through resuming, processing isperformed to display a home screen image on the display unit 12 undercontrol of the home screen image application 21. The home screen imagethat had been started through resuming is in an unlocked state where theuser can perform an operation input.

On the other hand, when the system is woken up from the suspend state inthe mobile terminal device 10 according to the present embodiment,operations of the units of the hardware layer are directly controlledfrom the middleware layer without going through the HMI layer. Forexample, in the mobile terminal device 10, the camera middleware 32 isdirectly controlled with the quick start 31 without going through anoperation of the camera application 22, which allows immediate imagingprocessing. Likewise, in the mobile terminal device 10, the multimediamiddleware 33 is directly controlled with the quick start 31 withoutgoing through an operation of the music reproduction application 23,which allows immediate music reproduction processing. Thus, in themobile terminal device 10, a shortcut function is achieved toimmediately execute various functions without going through controlperformed in the public API.

[4. Exemplary Scans Performed for Touch Sensor]

Here, exemplary scans that are performed for the touch sensor 71 todetect that an operation input is performed by a finger touching theoperation surface of the touch panel 7 will be described with referenceto FIGS. 4A, 4B, 5A, 5B, 6A, and 6B.

FIGS. 4A and 4B are explanatory diagrams illustrating examples of ausual scan and an intermittent scan. FIG. 4A illustrates an exemplaryusual scan, and FIG. 4B illustrates an exemplary intermittent scan. InFIGS. 4A and 4B, the horizontal axes indicate time (t), and the verticalaxes indicate the magnitude of a scan current.

The usual scan illustrated in FIG. 4A is performed when the mobileterminal device 10 is not in the suspend state. During the usual scan,the surface of the touch sensor 71 is scanned at a scan rate of 60 Hz,for example, and the response is increased to send a result back to anoperation input from the user. However, since scans are performed withan increased frequency during the usual scan, the power consumption ofthe touch panel is also increased.

The intermittent scan illustrated in FIG. 4B is performed when themobile terminal device 10 is in the suspend state. At that time, thetouch sensor 71, which is used as an input detection unit, decreases theexecution frequency of processing arranged to detect an input operationperformed for the touch panel when no input operation performed for thetouch panel is detected over a specified time period.

During the intermittent scan, the surface of the touch sensor 71 isscanned at a scan rate of 10 Hz, for example, and a sleep period wherethe various functions of the mobile terminal device 10 are stopped isprovided between each scan. Therefore, the power consumption of theintermittent scan is lower than that of the usual scan. Further, anoperation input from the user is performed with a decreased frequency inthe suspend state. Accordingly, it is advisable to detect the tap ordrag operation when those operations are performed with the minimumfrequency. The intermittent scan causes no operability discomfort solong as it is arranged that the usual scan is performed after detectingan operation input from the user and performing the resume processing.

FIGS. 5A and 5B are explanatory diagrams illustrating exemplarycapacitance values that are detected with the touch sensor 71 on theoperation surface during a touch operation performed in a usual state.FIG. 5A illustrates exemplary capacitance values that are obtainedduring a touch operation performed in the usual state, and FIG. 5Billustrates exemplary capacitance values that are displayed on athree-dimensional graph. Incidentally, the following description isprovided on premises that the short side of the touch panel 7 isdetermined to be X, the long side is determined to be Y, and thecoordinates of the touch panel 7 are equivalent to those of the touchsensor 71 in FIGS. 5A, 5B, 6A, and 6B.

As illustrated in FIG. 5A, a finger touches the position of coordinates(X, Y)=(3, 14) of the touch panel 7. At that time, capacitance values(90 to 100) near the position of coordinates (3, 14) are higher thancapacitance values (7 to 10) that are detected at different positions.Consequently, the touch sensor 71 detects that an operation input isperformed on the touch panel 7. According to the stereoscopic graph ofthe capacitance values, which is illustrated in FIG. 5B, it is evidentthat a capacitance value obtained at the position of coordinates (3, 14)is conspicuous. It is advisable to provide a specified threshold value(e.g., 50) for the capacitance value, and detect a touch operation whenthe capacitance value exceeds the threshold value.

FIGS. 6A and 6B are explanatory diagrams illustrating exemplarycapacitance values that are detected with the touch sensor 71 on theoperation surface in the state where the mobile terminal device 10 is ina pocket, etc. of the user. FIG. 6A illustrates exemplary capacitancevalues that are obtained when the mobile terminal device 10 is put intothe pocket, etc. of the user, and FIG. 6B illustrates an example wherethe capacitance values are displayed on a three-dimensional graph.

As illustrated in FIG. 6A, the touch sensor 71 outputs high capacitancevalues (40 to 80) over the entire face when the mobile terminal device10 is put into the pocket, etc. of the user, because a human bodytouches over the entire face of the touch panel 7 via cloth, etc. Atthat time, since the high capacitance values are detected over theentire face of the touch panel 7, the touch sensor drive unit 72determines that no intentional operation input is performed by the user,even though an object touches the touch panel 7 for some reason.According to the stereoscopic graph of the capacitance values, which isillustrated in FIG. 6B, it is evident that high capacitance values aredetected over the entire face of the touch sensor 71.

Thus, the touch sensor drive unit 72 detects nothing as a touchoperation when capacitance values are detected over the entire face ofthe touch sensor 71, so as not to cause the mobile terminal device 10 tomalfunction in the suspend state. Then, the malfunction is avoided byperforming control so that specified processing is executed when thetouch sensor 71 detects capacitance values for gesture operations thatare performed in specific patterns which will be described later.

[5. Exemplary Gesture Operations]

Next, exemplary gesture operations that are performed according to thepresent embodiment will be described with reference to FIGS. 7A, 7B, 8A,and 8B.

[5-1. Exemplary Gesture Operations Performed to Start Home Screen Image]

FIGS. 7A and 7B are explanatory diagrams illustrating an example where ahome screen image is displayed from the suspend state through a firstgesture operation. FIG. 7A illustrates an example of the first gestureoperation performed on the touch panel 7 by the user, and FIG. 7Billustrates an exemplary home screen image.

In FIGS. 7A and 7B, an example where the home screen image is started bythe user touching the operation surface of the touch panel 7 in place ofprocessing performed in the past to start a home screen image from thesuspend state by pressing the power key of the mobile terminal device100.

Incidentally, even though the mobile terminal device 10 is in thesuspend state, intermittently scanning the surface of the touch panel 7as illustrated in FIG. 4B allows for continuously performing the touchdetection processing.

The first gesture operation, which is an exemplary specified inputoperation, is an operation achieved by sweeping the touch panel 7 in a Zshape (see FIG. 7A) when the mobile terminal device 10 is in the suspendstate. When the touch sensor 71 detects that the first gesture operationis performed on the touch panel 7 in the suspend state, the touch sensordrive unit 72 displays a specified screen image on the display unit 12.

At that time, the mobile terminal device 10 starts resuming from thesuspend state, and starts the home screen image (see FIG. 7B). Since thehome screen image can be directly started from the suspend state withoutinputting a password, the user can immediately select a menu from thehome screen image through an easy operation.

[5-2. Exemplary Gesture Operation Performed to Execute Shooting]

FIGS. 8A and 8B are explanatory diagrams illustrating an example where asecond gesture operation is performed to start an imaging function fromthe suspend state and shoot a picture. FIG. 8A illustrates an examplewhere the second gesture operation is performed on the touch panel 7 bythe user, and FIG. 8B illustrates an example where an image obtainedthrough picture shooting is displayed on the display unit 12.

Since the touch sensor drive unit 72 can detect a specific pattern madeon the operation surface on the touch panel 7, operations of variousapplications, other than the function of starting the home screen imagefrom the suspend state as illustrated in FIGS. 7A and 7B, can becombined with specific patterns that are made on the operation surface.A function illustrated in FIGS. 8A and 8B achieves capturing a camerathrough image shot by the imaging unit 14 by performing a drag operationachieved by drawing a circle with a finger on the operation surface ofthe touch panel 7 when the mobile terminal device 10 is in the suspendstate.

The second gesture operation, which is an exemplary specified inputoperation, is an operation achieved by sweeping the touch panel 7 in acircular shape (see FIG. 8A) when the mobile terminal device 10 is inthe suspend state. When the touch sensor 71 detects that the secondgesture operation different from the first gesture operation isperformed on the touch panel 7 in the suspend state, the touch sensordrive unit 72 causes the imaging drive unit 13 to start an operation ofthe imaging unit 14. Then, the touch sensor drive unit 72 causes thedisplay unit 12 to display an image obtained by imaging performed withthe imaging unit 14.

At that time, the mobile terminal device 10 starts resuming from thesuspend state, starts the imaging drive unit 13, and shoots a subjectthrough the imaging unit 14 without starting the camera application 22(see FIG. 8B). An image of the subject, which is obtained through theimaging, is displayed on the display unit 12. Since an operation of theimaging unit 14 is not executed after going through an imaging menu fromthe home screen image, the user can immediately perform imaging atdesired time for taking an image of the subject.

Next, an exemplary comparison between the sequence of an example whereshooting is performed by starting the camera application 22 through aknown operation and that of an example where high-speed shooting isperformed through a function according to the present embodiment will bedescribed with reference to FIGS. 9 and 10. Incidentally, the mobileterminal device 10 according to the present embodiment can perform boththe known operation and the function according to the presentembodiment.

FIG. 9 is a sequence diagram illustrating exemplary processing executedto start the camera application 22 and perform shooting through theknown operation.

The known operation allows for starting the camera application 22 afterthe system of the mobile terminal device 10 is woken up after the userpresses the power key in the suspend state. Then, the shootingprocessing is executed after the camera application 22 is started.

In the following description, the mobile terminal device 10 being in thesuspend state or shifting to various modes is expressed by the controlunit 3 functioning as a system illustrated in the sequence diagram.

First, it is assumed that the system is in the suspend state (step S1).When the power key included in the operation keys 8 is pressed when thesystem is in the suspend state, processing performed to resume thesystem is started (step S2).

At that time, the control unit 3 outputs a resume instruction to thetouch sensor drive unit 72 (step S3), the touch sensor drive unit 72performs resume processing for the touch sensor 71, and the touch sensor71 waits for an operation performed to detect an input operation (stepS4). Then, an unillustrated camera icon provided to start the cameraapplication 22 is displayed on the display unit 12 unillustrated in FIG.9.

Further, the control unit 3 outputs a resume instruction to the imagingdrive unit 13 (step S5), and the imaging drive unit 13 performs resumeprocessing for the imaging element 14 a, and waits for an input of animaging operation (step S6). After resuming processing performed foreach unit is finished, the system enters a wakeup state where a responseis sent back to an operation input from the user (step S10).

Upon detecting that an input operation achieved with a touch isperformed at the position where the camera icon is displayed (step S7),the touch sensor 71 notifies the touch sensor drive unit 72 of thecoordinates where the touch is detected (step S8). Further, the touchsensor drive unit 72 notifies the control unit 3 of the coordinates(step S9).

Upon being informed that the input operation is performed for the cameraicon through the notification from the touch sensor drive unit 72, thecontrol unit 3 outputs an instruction to start the camera application 22to the camera application 22 (step S11). Upon receiving the startinstruction, the camera application 22 initializes the camera middleware32 (step S12). Then, the camera middleware 32 outputs an instruction toperform power-on processing and initialization processing to the imagingdrive unit 13 (step S13). Accordingly, the imaging drive unit 13 isenergized and the initialization processing is performed.

Further, the imaging drive unit 13 outputs an instruction to performpower-on processing and initialization processing to the imaging element14 a (step S14). Accordingly, the imaging element 14 a is energized andthe initialization processing is performed. Then, a shooting iconprovided to specify a shooting operation is displayed on the displayunit 12.

Upon detecting that an input operation achieved with a touch isperformed at the position where the shooting icon is displayed (stepS15), the touch sensor 71 notifies the touch sensor drive unit 72 of thecoordinates where the touch is detected (step S16). Further, the touchsensor drive unit 72 notifies the control unit 3 of the coordinates(step S17).

Upon being informed that the input operation is performed for theshooting icon through the notification from the touch sensor drive unit72, the control unit 3 notifies the camera application 22 of the touchedcoordinates (step S18). Upon receiving the touched coordinates, thecamera application 22 outputs an instruction to start shootingprocessing to the camera middleware 32 (step S19).

The camera middleware 32 instructs the imaging drive unit 13 to performprocessing to capture a shot image (step S20), and the imaging driveunit 13 performs the processing to capture the shot image based on animage signal output from the imaging element 14 a (step S21). The imagesubjected to the capture processing is displayed on the display unit 12.

Thus, according to known processing, the user firstly presses the powerbutton to cancel the suspend state, searches and selects the camera iconon the home screen image displayed through the resume processing, andperforms shooting with the shooting icon. Therefore, many procedures areperformed from the suspend state until the shooting is started, whichleads to complicated operations.

FIG. 10 is a sequence diagram illustrating exemplary processing executedto start the camera application 22 and perform shooting through anoperation performed according to the present embodiment.

First, it is assumed that the system is in the suspend state (step S31).When the mobile terminal device 10 is in the suspend state, theintermittent scan is performed for the touch sensor 71 to wait for thedetection of a touch operation in the low-power consumption mode wherethe consumption power is kept low (step S32).

When an operation input is performed on the touch panel 7 through thesecond gesture operation instructing to start performing shooting, thetouch sensor 71 detects that the operation input is performed (stepS33). At that time, the touch sensor 71 outputs a particular interruptinstruction issued to wake up the system to the touch sensor drive unit72 (step S34).

Incidentally, when the control unit 3 is in the suspend state, noprocessing is performed upon receiving a usual interrupt instruction.However, the particular interrupt instruction instructs the control unit3 to start performing resume processing. Therefore, upon receiving theparticular interrupt instruction, the control unit 3 starts performingthe resume processing and performs processing to cause a transition fromthe suspend state to a wakeup state.

The touch sensor drive unit 72 outputs an instruction to start resumeprocessing for the system to the control unit 3 as the particularinterrupt instruction (step S35). Accordingly, the control unit 3 startsperforming the resume processing for the system (step S36).

Further, the touch sensor drive unit 72 outputs an instruction toperform wakeup processing for the camera middleware 32, and aninstruction specifying the start of capture processing to the quickstart 31 (step S37). Upon receiving the instructions from the touchsensor drive unit 72, the quick start 31 outputs the instruction toperform the wakeup processing for the camera middleware 32 to the cameramiddleware 32, and outputs the instruction specifying the start of thecapture processing (step S38).

After performing the wakeup processing, the camera middleware 32 outputsan instruction to perform power-on processing and initializationprocessing to the imaging drive unit 13 (step S39). Accordingly, theimaging drive unit 13 is energized and the initialization processing isperformed.

Further, the imaging drive unit 13 outputs an instruction to performpower-on processing and initialization processing to the imaging element14 a (step S40). Accordingly, the imaging element 14 a is energized andthe initialization processing is performed. Then, the shooting iconprovided to specify the shooting operation is displayed on the displayunit 12.

The resume processing performed for the system is finished, and thecontrol unit 3 enters the wakeup state (step S41). Next, the cameramiddleware 32 instructs the imaging drive unit 13 to perform processingto capture a shot image (step S42). At that time, the camera middleware32 outputs an instruction to start the camera application 22 (step S43).

Then, the imaging drive unit 13 performs the processing to capture theshot image based on an image signal output from the imaging element 14 a(step S44). The image subjected to the capture processing is displayedon the display unit 12. Then, for the image displayed on the displayunit 12, an operation menu, etc. provided due to the start of the cameraapplication 22 is displayed on the display unit 12.

Thus, processing or software executed through the second gestureoperation is limited. Then, the camera application 22 is started aftercapturing the image immediately after detecting the gesture operation,which minimizes the time that elapses before the shooting is started.Accordingly, it becomes possible to shoot the subject at the moment theuser wishes to shoot, and eliminate the user's complaint about missingtiming by taking too much time on startup for performing shootingthrough the mobile terminal device 10 which is in the suspend state.

[5-3. Exemplary Gesture Operation Executed to Perform MusicReproduction]

Next, an exemplary gesture operation executed to perform the musicreproduction will be described with reference to FIGS. 11A, 11B, and 12.

FIGS. 11A and 11B are explanatory diagrams illustrating an example wherea music reproduction function is started from the suspend state and amusic file is reproduced through a third gesture operation. FIG. 11Aillustrates an example of the third gesture operation performed on thetouch panel 7 by the user, and FIG. 11B illustrates an example wheremusic is emitted by reproducing the music file.

The music reproduction function described here reduces the time thatelapses before the music reproduction is started when an operation isperformed on the touch panel 7 through a specific gesture. Incidentally,as being operated with low power consumption, the mobile terminal device10 is in an early suspend state where the display unit 12 is in thesuspend state and no image is displayed.

The third gesture operation, which is an exemplary specified inputoperation, is an operation achieved by sweeping the touch panel 7 in anN shape (see FIG. 11A) when the mobile terminal device 10 is in thesuspend state. When the touch sensor 71 detects that the third gestureoperation is performed on the touch panel 7 in the suspend state, thetouch sensor drive unit 72 causes the voice processing to startperforming voice processing, and causes a sound emitting unit to emit avoice.

At that time, the mobile terminal device 10 starts performing resumingfrom the suspend state, starts the voice processing unit 4, and emitsreproduced music through the speaker 5 (see FIG. 11B). When the displayunit 12 is started in the early suspend state, various informationrelating to the music which is being reproduced (e.g., the track name,the singer's name, and so forth) is not displayed on the display unit12. However, it may be arranged that the display unit 12 is started andthe various information relating to the music which is being reproducedis displayed on the display unit 12.

FIG. 12 is a sequence diagram illustrating exemplary processing executedto start the music reproduction application 23 and reproduce musicthrough an operation according to the present embodiment.

First, it is assumed that the system is in the suspend state (step S51).When the mobile terminal device 10 is in the suspend state, theintermittent scan is performed for the touch sensor 71 to wait for thedetection of a touch operation in the low-power consumption mode (stepS52).

When an operation input is performed on the touch panel 7 through agesture performed to restart the music reproduction (the third gestureoperation), the touch sensor 71 detects that the operation input isperformed (step S53). At that time, the touch sensor 71 outputs aparticular interrupt instruction to the touch sensor drive unit 72 (stepS54).

The touch sensor drive unit 72 outputs an instruction to start resumeprocessing for the system and start the early suspend state to thecontrol unit 3, as a particular interrupt instruction (step S55).Consequently, the control unit 3 starts performing the resume processingfor the system (step S56).

Further, the touch sensor drive unit 72 outputs an instruction toperform processing to restart the music reproduction of the multimediamiddleware 33 to the quick start 31 (step S57). Upon receiving theinstruction from the touch sensor drive unit 72, the quick start 31outputs the instruction to perform the processing to restart the musicreproduction of the multimedia middleware 33 to the multimediamiddleware 33 (step S58).

After that, the multimedia middleware 33 outputs an instruction toperform power-on processing and initialization processing to the voiceprocessing unit 4 (step S59). Accordingly, the voice processing unit 4is energized and the initialization processing is performed.

Further, the voice processing unit 4 outputs an instruction to performthe power-on processing and the initialization processing to the speaker5 (step S60). Accordingly, the speaker 5 is energized, theinitialization processing is performed, and reproduced music is emitted.

After that, when the resume processing performed for the system isfinished, the system enters the early suspend state (step S61). At thattime, an off state is achieved so that no image is displayed on thedisplay unit 12. Then, when the system enters the early suspend state,the multimedia middleware 33 transmits a start notification indicatingthat the music reproduction is restarted to the music reproductionapplication 23 (step S62). Upon receiving the notification, the musicreproduction application 23 is started.

The above-described mobile terminal device 10 according to the firstembodiment allows for immediately executing a specific function in thesuspend state when an operation input is performed on the touch panel 7through a specific gesture operation. Here, the modules that areprovided in the middleware layer directly output instructions andinstruct the units that are included in the hardware layer to startperforming operations without going through the various applicationsthat are provided in the HMI layer (see FIG. 3). Therefore, the user isallowed to execute various functions that are allocated to specificgesture operations without waiting for the time-consuming start of thevarious applications that are provided in the HMI layer, which leads toincreased operability.

Further, when the first gesture operation is performed on the touchpanel 7 of the mobile terminal device 10 which is in the suspend state,the home screen image is started without operating the operation key 8.When each user can separately set the first gesture operation to startthe home screen image, the password input can be omitted by using thefirst gesture operation which is only known by the user in place of thepassword input.

Further, when the second gesture operation is performed on the touchpanel 7, shooting can be started by immediately starting the imagingdrive unit 13 and the imaging unit 14. In that case, the shooting canalso be performed without waiting for the start of the cameraapplication 22, which takes much time before the start, which avoidslosing the imaging timing.

Further, when the third gesture operation is performed on the touchpanel 7, the music reproduction can be started by immediately startingthe music reproduction unit 4. By restarting the reproduction from somemidpoint in a track, where the music reproduction had been stopped, theuser is allowed to reproduce and immediately enjoy music he had beenlistening to until the use of the mobile terminal device 10 was stopped.

Further, when the mobile terminal device 10 is in the suspend state, thepower consumption can be reduced by performing the intermittent scan.Further, a distribution of capacitance values is obtained when the areawhere the touch sensor 71 detects an operation input is part of or theentire face of the touch sensor 71, which allows for separatelydetermining whether or not the user intentionally performed theoperation input. Therefore, for an unintentional operation input whichis performed when a human body indirectly touches the touch panel 7 ofthe mobile terminal device 10 which is put into a pocket by the user, itbecomes possible to avoid erroneously determining that an operationinput is done.

Incidentally, for the mobile terminal device 10 according to the firstembodiment, exemplary applications regarding the startup of the homescreen image, the shooting, and the music reproduction have beendescribed. Additionally, however, any application may be used so long asit is desired to be immediately started for use. For example, a mapapplication illustrating the current value, etc., or an application thatlogs into SNS (Social Networking Service) or makes access to a blog siteor a news site may be used.

Second Embodiment

[6. Example where Operation is Performed in Concert with Another Device]

Next, the mobile terminal device 10 according to a second embodiment ofthe present disclosure will be described with reference to FIG. 13.

The mobile terminal device 10 according to the second embodimentimmediately achieves a function used by a user not only by variousgesture operations that are operation-input to the touch panel 7, butalso by combining a sensor value output from the sensor unit 18.

When the touch sensor 71 detects that a specified input operation isperformed on the touch panel 7 in the suspend state, the touch sensordrive unit 72 selects a single controlled unit from among plural typesof controlled units based on the sensor value, and causes the selectedcontrolled unit to execute a specified operation.

FIG. 13 is a flowchart illustrating exemplary processing performed toexecute various functions by combining a gesture operation with anacceleration value detected with the acceleration sensor.

It is assumed that the mobile terminal device 10 is in the suspendstate. Then, it is also assumed that the touch sensor 71 performs theintermittent scan (see FIG. 4B), and as for a sensor value (accelerationvalue) from the acceleration sensor included in the sensor unit 18, thesensor value is output at the same scan rate as that of the intermittentscan.

First, the touch sensor drive unit 72 determines whether or not a fourthgesture operation achieved to start a shortcut function executingvarious functions without going through an application is performed onthe touch panel 7 (step S71). Here, since the fourth gesture operation,which is an exemplary specified input operation, is not used to select adesired function to be executed, it may be equivalent to or differentfrom any one of the above-described first to third gesture operations.When the fourth gesture operation is not detected, the determinationprocessing of step S71 is repeated.

Next, it is determined whether or not the touch sensor 71 detects afifth gesture operation (e.g., a double tap) achieved to perform a quickoperation (step S72). The fifth gesture operation includes, for example,an operation achieved by tapping the touch panel 7 twice.

When the fifth gesture operation is not performed, resume processing isstarted for the system (step S73). The resume processing described hereis an alternative to the processing performed to start a home screenimage by pressing the power key, for example. Then, the display unit 12is caused to display an unlocked home screen image and operations arecontinued in the state where the system wakes up.

When the fourth gesture operation is performed at step S72, the quickstart 31 detects the orientation of the mobile terminal device 10 basedon the sensor value received from the acceleration sensor (step S74).Then, it is determined whether or not the orientation is the lateraldirection of the mobile terminal device 10 (step S75). Here, the mobileterminal device 10 being oriented in the lateral direction indicatesbeing at rest in the state where the long side of the cabinet 19 of themobile terminal device 10 is almost parallel to the horizontal. Further,the mobile terminal device 10 being oriented in a vertical directionindicates being at rest in the state where the long side of the cabinet19 of the mobile terminal device 10 is almost parallel to a verticalline.

Upon determining that the mobile terminal device 10 is oriented in thelateral direction, the quick start 31 instructs each unit to start theabove-described processing illustrated in FIG. 10. Then, the imagingunit 14 performs capture processing for an image of the subject, whichis obtained through imaging, for display on the display unit 12 (stepS76). After that, the camera application 22 is started (step S77), andthe operations of the imaging unit 14 are continued in the state wherethe system wakes up.

Upon determining that the mobile terminal device 10 is oriented in thevertical direction, the quick start 31 instructs each unit to start theabove-described processing illustrated in FIG. 12. Then, the voiceprocessing unit 4 causes the speaker 5 to emit music of whichreproduction is restarted from the spot where the music reproduction isstopped (step S78). After that, the music reproduction application 23 isstarted, and the operations of the voice processing unit 4 are continuedin the state where the system wakes up.

The above-described mobile terminal device 10 according to the secondembodiment determines the type of an operation input performed for themobile terminal device 10 based on a sensor value (acceleration value)from the acceleration sensor included in the sensor unit 18, in additionto a gesture operation performed on the touch panel 7. At that time,starting the music reproduction in accordance with a gesture operationperformed in the state where the mobile terminal device 10 is held inthe vertical direction, and immediately starting performing imaging withthe imaging unit 14 in accordance with a gesture operation performed inthe state where the mobile terminal device 10 is held in the lateraldirection are achieved.

Further, when the sensor unit 18 includes a gyro sensor, the directionin which the mobile terminal device 10 is placed may be determined basedon a sensor value transmitted from the gyro sensor. Likewise, theorientation of the mobile terminal device 10 may be determined throughthe use of a magnetic field sensor. Further, at step S72, a transitionto the unlocked screen image is caused when the tap operation isperformed. However, the processing of step S72 may be omitted. In thatcase, the orientation of the mobile terminal device 10 can be detectedimmediately after the shortcut function is started.

Further, during branch processing of step S72 of FIG. 13, it may bearranged that the unlocked screen image is displayed after the doubletap is detected, and a specified application is started after a passwordis input by the user.

Third Embodiment

[7. Example where Application is Started Through Two-Step GestureOperation]

Next, the mobile terminal device 10 according to a third embodiment ofthe present disclosure will be described with reference to FIGS. 14A,14B, 14C, 14D, and 14E.

The mobile terminal device 10 according to the third embodimentimmediately achieves a specific function when an operation input isperformed on the touch panel 7 in two steps.

FIGS. 14A, 14B, 14C, 14D, and 14E are explanatory diagrams illustratingan exemplary operation performed to start an application through atwo-step gesture operation. FIG. 14A illustrates an exemplary first-stepgesture operation, FIGS. 14B and 14C illustrate exemplary second-stepgesture operations, and FIGS. 14D and 14E illustrate exemplarily startedapplications.

In the mobile terminal device 10 according to the third embodiment, anapplication is immediately executed not only through the gestureoperation performed at the first step. The gesture operation performedat the first step is only a preliminary operation, and the applicationis executed through the gesture operation performed at the second step.

Then, when the touch sensor 71 detects that different gestures areperformed plural times on the touch panel 7 in predetermined order inthe suspend state, the touch sensor drive unit 72 causes a controlledunit associated with a different input operation to execute a specifiedoperation.

First, it is assumed that the user performs an input operation in thestate where the long side of the cabinet 19 of the mobile terminaldevice 10 is almost parallel to the horizontal. For example, the gestureoperation of the first step is an operation achieved by the usersweeping the touch panel 7 in a Z shape when the mobile terminal device10 is in the suspend state (see FIG. 14A).

For starting a Web browser, the user performs an operation by sweeping afinger from near the center of the touch panel 7 toward the leftdirection in a straight line (see FIG. 14B) as the gesture operation ofthe second step. After the operation, the mobile terminal device 10connects to the Internet and the Web browser is started so that anaccessed site can be displayed on the display unit 12 (see FIG. 14D).

On the other hand, for starting a gallery application of a picture, theuser performs an operation by sweeping a finger from near the center ofthe touch panel 7 toward the right direction (see FIG. 14C) as thegesture of the second step. After the operation, the mobile terminaldevice 10 can start the gallery application and display a shot image orvideo read from the storage unit 15 on the display unit 12 as athumbnail (see FIG. 14E).

The above-described mobile terminal device 10 according to the thirdembodiment allows for starting plural types of applications by combiningdifferent gestures in two steps. As stated above, the applications forstarting include the Web browser, the gallery application, etc. When thegallery application is started, a specific page may be displayed at theoutset. Further, when the Web browser is started, a search page arrangedto make a word search may be displayed so that the user can immediatelymake the word search. Further, a specific URL may be specified as anargument (parameter) passed to an application on startup.

Further, the gesture operation, which exemplarily includes the operationachieved by sweeping the finger caused to touch the touch panel 7 in a Zshape and that achieved by sweeping in the horizontal direction, may becombined with a different gesture operation including, for example, anoperation achieved by tapping or sweeping the touch panel 7 in acircular shape. Further, an application may be started by performing atwo-step gesture operation while holding the mobile terminal device 10in a vertical direction. Further, gesture operations may be combined inthree or more steps, even though the combination leads to complicatedoperations.

Fourth Embodiment

[8. Exemplary Assistance Function Started when Gesture Operation Ends inFailure]

Next, the mobile terminal device 10 according to a fourth embodiment ofthe present disclosure will be described with reference to FIGS. 15A,15B, 15C, 15D, and 16.

The mobile terminal device 10 according to the fourth embodimentdisplays a guide arranged to assist a gesture operation when the gestureoperation ends in failure, which makes it impossible to start anapplication.

FIGS. 15A, 15B, 15C, and 15D are explanatory diagrams illustratingexemplary assistance processing performed when a gesture operation endsin failure. FIG. 15A illustrates an example where a gesture operationends in failure, and FIG. 15B illustrates an example where the guide isdisplayed on the display unit 12. Further, FIG. 15C illustrates anexample where the gesture operation is performed again along the guide,and FIG. 15D illustrates an example where an application is successfullystarted.

The above-described gesture operations that are performed according tothe first to third embodiments are performed for the mobile terminaldevice 10 which is in the suspend state. However, since nothing isdisplayed on the display unit 12 when the mobile terminal device 10 isin the suspend state, the gesture operations are often performed atwrong positions, which makes it impossible to start an application (seeFIG. 15A).

At that time, when the touch sensor 71 detects that a specified inputoperation is similar to a stipulated input operation, the touch sensordrive unit 72 causes the display unit 12 to display a guide representingthe stipulated input operation. Then, when the touch sensor 71 detectsthat an input operation performed again on the touch panel 7 inaccordance with the guide is the stipulated input operation, acontrolled unit associated with the stipulated input operation isinstructed to execute a specified operation.

Specifically, the mobile terminal device 10 according to the fourthembodiment displays a guide 75 on the display unit 12 as a function ofcomplementing a gesture operation (see FIG. 15B). For example, when agesture operation that can be detected with the touch sensor 71 is anoperation achieved by sweeping the touch panel 7 in a Z shape near thecenter of the touch panel 7, the guide 75 displays a Z-shaped effectivearea on the display unit 12.

At that time, the user performs a gesture operation by referring to theguide 75 displayed on the display unit 12 (see FIG. 15C). When asweeping operation is performed by a finger which is kept in contactwith the touch panel 7 in the effective area of the guide 75, the touchsensor 71 recognizes that the gesture operation is performed withincreased reliability. Further, the width of the displayed guide 75 iswider than that of the finger. Therefore, the user can perform thegesture operation while keeping the finger from deviating from the guide75. When the gesture operation is recognized, an application issuccessfully started, and an application startup is performed (see FIG.15D).

Further, for displaying the guide 75, the resume processing performedfor the mobile terminal device 10 operates. However, the operation onlycauses the guide 75 to be displayed without displaying a locked screenimage. Further, for determining that a gesture operation ends infailure, it is necessary to make sure not to react in such a way as todeal with a gesture operation that ends in failure when a touchoperation is performed in the case where the mobile terminal device 10is put into a pocket, or a finger touches the touch panel 7 just alittle. This is because displaying the guide 75 consumes too much power.Therefore, it is desirable to make a comparison between a stipulatedgesture operation and an operation detected with the touch sensor 71,and display the guide 75 when a similarity between the operations isconfirmed.

FIG. 16 is a sequence diagram illustrating exemplary processingperformed to execute the assistance function.

First, it is assumed that the system is in the suspend state (step S81).When the mobile terminal device 10 is in the suspend state, theintermittent scan is performed for the touch sensor 71 to wait for thedetection of a touch operation in the low-power consumption mode (stepS82).

The touch sensor 71 detects that an operation similar to a gestureoperation that wakes up the system (referred to as “wakeup gesture” inthe drawing) is performed on the touch panel 7 (step S83). At that time,the touch sensor 71 outputs a particular interrupt instruction to thetouch sensor drive unit 72 (step S84).

The touch sensor drive unit 72 outputs an instruction to start resumeprocessing performed for the system and start performing the resumeprocessing to the control unit 3 as the particular interrupt instruction(step S85). Consequently, the control unit 3 starts performing theresume processing for the system (step S86).

When the control unit 3 starts performing the resume processing, thetouch sensor drive unit 72 outputs a guide display request issued todisplay the guide 75 to the quick start 31 (step S87). Upon receivingthe guide display request from the touch sensor drive unit 72, the quickstart 31 outputs the guide display request to the multimedia middleware33 (step S88). Here, the multimedia middleware 33 has a guide displayfunction provided to find a gesture operation similar to ahard-to-determine operation when the operation is performed on the touchpanel 7 and cause the display unit 12 to display the guide 75.

Upon receiving the guide display request from the touch sensor driveunit 72, the multimedia middleware 33 outputs an instruction to performthe guide display to the display processing unit 9 (step S89). Then, thedisplay processing unit 9 stores an image of the guide 75 in the framebuffer 11. The display unit 12 displays the guide 75 based on the imageread from the frame buffer 11 (step S90). After that, the control unit 3causes the system to enter the wakeup state (step S91), and waits untila gesture operation is input again. At that time, the system is in aguide display mode where the guide 75 is displayed on the display unit12.

When the touch sensor 71 detects a gesture operation that wakes up thesystem in the guide display mode (step S92), the touch sensor 71 outputsan instruction requesting that the system be woken up to the touchsensor drive unit 72 (step S93). An instruction to start the resumeprocessing performed for the system and start performing the resumeprocessing is output to the control unit 3 (step S94). Consequently, thecontrol unit 3 causes the system to enter the wakeup state (step S95).

After that, the touch sensor drive unit 72 outputs a guide displayrequest to display the guide 75 to the multimedia middleware 33 (stepS96). Further, an instruction to update and display a menu imageattained in the usual state on the display unit 12 is output from thesystem which had entered the wakeup state to the frame buffer 11 (stepS97). The frame buffer 11 performs processing to update the menu imagedisplayed on the display unit 12 (step S98).

The above-described mobile terminal device 10 according to the fourthembodiment is effective when a gesture operation that wakes up thesystem cannot be recognized due to a drag position drift, etc. In thatcase, when an operation similar to a stipulated gesture operation isperformed, the guide 75 is displayed on the display unit 12 so that anoperation is performed again. Then, a specified function is executedbased on a gesture operation performed along the guide 75. Consequently,since no image is displayed on the mobile terminal device 10 which is inthe suspend state, the user who had failed in a gesture operation isallowed to perform a gesture operation along the guide 75 at there-operation time so that an input operation can be performed withreliability.

Further, when a gesture operation ends in failure, the system is notfully woken up. However, the system changes to the guide display modeand continuously waits until a gesture operation is performed. Then,after detecting an appropriate gesture operation performed again, thesystem wakes up into the usual state. From then on, the screen imageupdate, operations, and so forth are performed in the usual state.

Further, it may be arranged that the locus of a gesture operation isdisplayed on the operation surface of the touch panel 7. The displayedlocus allows the user to easily recognize how the input operation endsin failure.

Fifth Embodiment

Next, the mobile terminal device according to a fifth embodiment of thepresent disclosure will be described with reference to FIG. 18 throughFIG. 28B.

FIG. 18 is a schematic diagram illustrating an exemplary externalconfiguration of a mobile terminal device according to a fifthembodiment of the present disclosure. In this embodiment, the touchpanel 7 includes plurality of panels, such as a front panel located on afront side of the mobile terminal device 10, side panel(s) located onlateral side(s) of the mobile terminal device 10 and/or a back panel(not shown) located on a back side of the mobile terminal device 10. Forexample, when an input operation is not detected by the front panel overa specified time period (e.g. 15 seconds), a frequency of detection(scan rate) by the front panel may be decreased anywhere from 60 Hz to 5Hz or even 0 Hz (e.g. deep sleep mode or power off mode) and a frequencyof detection (scan rate) by the side panels may be decreased from 60 Hzto 10 Hz (low power scanning mode). Thus, in this embodiment, thefrequency of detection by the front panel may be lower than that of sidepanels so that power consumption may be reduced.

FIG. 19A is an explanatory diagram illustrating an example where amobile terminal device is held by one hand on the lateral sides andexemplary object detected area of side panels according to the fifthembodiment of the present disclosure. FIG. 19B is a schematic diagramshow position points where objects are detected. In this example,processing circuitry (such as a CPU, ASCI, PLA, ect.) may execute aspecific operation (e.g. waking up the mobile terminal device) when apredetermined contact of an input operation on the side panelscorresponding to the specific operation is detected by an input detector(e.g. predetermined multiple locations of side panels are touchedsimultaneously by holding the mobile terminal device by one hand). Inthis example, a front panel located on the front side of the mobileterminal device is set to be deep sleep mode or power off mode so thatan input operation on the front panel is not detected. On the otherhand, a right side panel located on the right side of the mobileterminal device and a left side panel located on the left side of themobile terminal device are set to be low power scanning mode so that theright and left side panels can detect an input operation.

FIG. 20 is a sequence diagram illustrating an exemplary process flowperformed to wake up a mobile terminal device by detecting holding ofthe mobile terminal device by side panels according to the fifthembodiment of the present disclosure. Other than the touch sensor beinglocated on the right and left side of the mobile terminal device andholding of the mobile terminal device is detected by the right and leftside panels, many of the steps perform a similar function to thosedescribed in FIG. 16.

FIG. 21A is an explanatory diagram illustrating an example where amobile terminal device is held to shoot by two hands on the lateralsides. FIG. 21B is a schematic diagram showing exemplary object detectedareas of side panels according to the fifth embodiment of the presentdisclosure. In this example, processing circuitry may execute a specificoperation (e.g. starting a camera application and capturing image) whena predetermined contact of an input operation on the side panelscorresponding to the specific operation is detected by an input detector(e.g. predetermined multiple locations of the side panels are touchedsimultaneously by holding the mobile terminal device to shoot aphotograph by two hands).

FIG. 22 is a sequence diagram illustrating an exemplary process flowexecuted to start the camera application and perform shooting through anoperation detected by side panels according to the fifth embodiment ofthe present disclosure. Other than the touch sensor is located on theright and left side of the mobile terminal device and holding of themobile terminal device to shoot is detected by the right and left sidepanels, many of the steps perform a similar function to those describedin FIG. 10.

FIG. 23A is an explanatory diagram illustrating an example where amobile terminal device is held by one hand avoiding touching side panelsand FIG. 23B is a schematic diagram showing exemplary object detectedareas of side panels according to the fifth embodiment of the presentdisclosure. In this example, a mobile terminal device is held bytouching only a front panel which is set to be deep sleep mode or poweroff mode, avoiding touching side panels which are set to be low powerscanning mode. Therefore, no input operation is detected by the frontpanel and an unexpected operation of the mobile terminal device may beavoided.

FIG. 24A is an explanatory diagram illustrating an example where amobile terminal device is held by one hand and FIG. 24B is a schematicdiagram showing exemplary object detected areas of a front panelaccording to the fifth embodiment of the present disclosure. In thisexample, a processing circuitry may execute a specific operation (e.g.waking up the mobile terminal device) when a predetermined contact of aninput operation on the front panel corresponding to the specificoperation is detected by an input detector (e.g. predetermined multiplelocations of the front panel are touched simultaneously by holding themobile terminal device by one hand).

FIG. 25 is a sequence diagram illustrating an exemplary process flowperformed to wake up a mobile terminal device by detecting holding ofthe mobile terminal device by a front panel according to the fifthembodiment of the present disclosure. Other than holding of the mobileterminal device is detected not by gesture, but by detectingsimultaneous touch of predetermined multiple locations of the frontpanel, many of the steps perform a similar function to those describedin FIG. 16.

FIG. 26A is an explanatory diagram illustrating an example where amobile terminal device is held to shoot by two hands and FIG. 26B is aschematic diagram showing exemplary object detected areas of a frontpanel according to the fifth embodiment of the present disclosure. Inthis example, a processing circuitry may execute a specific operation(e.g. starting a camera application and capturing image) when apredetermined contact of an input operation on the front panelcorresponding to the specific operation is detected by an input detector(e.g. predetermined multiple locations of the front panel are touchedsimultaneously by holding the mobile terminal device to shoot aphotograph by two hands).

FIG. 27 is a sequence diagram illustrating an exemplary process flowexecuted to start the camera application and perform shooting through anoperation detected by a front panel according to the fifth embodiment ofthe present disclosure. Other than holding of the mobile terminal deviceis detected not by gesture, but by detecting simultaneous touch ofpredetermined multiple locations of the front panel to shoot, many ofthe steps perform a similar function to those described in FIG. 10.

FIG. 28A is an explanatory diagram illustrating an example where amobile terminal device is held by one hand avoiding touching a frontpanel and FIG. 28B is a schematic diagram showing exemplary objectdetected areas of a front panel according to the fifth embodiment of thepresent disclosure. In this example, as a mobile terminal device is heldby avoiding touching a front panel which is set to be a low powerscanning mode, no input operation is detected by the front panel and anunexpected operation of the mobile terminal device may be avoided.

In each of the described embodiments, a user may change a predeterminedcontact of an input operation from a touch panel. For example, a usermay change a predetermined gesture or predetermined multiple locationsto be touched simultaneously to start a camera application to a newcontact by changing a setting from the touch panel. Based on the changeof the setting by the user, processing circuitry changes thepredetermined contact of the input operation required to execute aspecific operation to the new predetermined contact.

Incidentally, the above-described functions according to the first tofourth embodiments may be used separately or arbitrarily combined withone another. Further, the type of a gesture operation is not limited tothe above-described embodiments so that a combination of the pinchoperation and the tap operation may be provided. Thus, the combinationof a gesture operation, a function, and so forth may be appropriatelychanged based on specifications of the mobile terminal device 10.

Further, a series of processings according to the above-describedembodiments is often executed by software. In that case, the executionis enabled by a computer having specifically designed hardware where aprogram constituting the software is installed, or a computer where aprogram achieved to execute various functions is installed. For example,a program constituting desired software may be installed in ageneral-purpose computer, etc. for the execution.

Further, a storage medium storing program code of software forimplementing the functions of the above-described embodiments may besupplied to a system or an apparatus. Further, it is needless to saythat a computer (or a control device such as a CPU) of the system or theapparatus may read and execute the program code stored in the storagemedium so that the functions are implemented.

The storage medium provided for supplying the program code may include,for example, a flexible disk, a hard disk, an optical disk, amagneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a non-volatilememory card, a ROM, etc.

Further, the functions of the above-described embodiments areimplemented by the computer reading and executing the program code.Additionally, an OS or the like running on the computer executes part ofor the entire actual processing based on instructions from the programcode. The functions of the above-described embodiments may beimplemented by the processing, which constitutes another embodiment.Further, the processings according to the present embodiments may beconstituted by hardware instead of being executed by software as statedabove.

Further, it is taken for granted that the present disclosure is notlimited to the above-described embodiments, but may be applied andmodified in different and various ways without leaving the spirit of thepresent disclosure, which is written in the claims. To this end, itshould be understood by those skilled in the art that variousmodifications, combinations, and other exemplary embodiments may occurdepending on design and/or other factors insofar as they are within thescope of the claims or the equivalents thereof, as a matter of course.

Incidentally, the present disclosure may be configured as below.

(1)

A mobile terminal device including:

a touch panel where a specified input operation is performed with anindicator,

an input detection unit configured to detect a pattern of the inputoperation based on a position of the indicator on the touch panel, anddecrease a frequency of execution of processing arranged to detect theinput operation performed on the touch panel when the input operationperformed on the touch panel is not detected over a specified timeperiod, and

an operation determination unit configured to instruct a controlled unitdetermined to be an operation target based on the pattern of the inputoperation to execute a specified operation when the pattern of the inputoperation is detected in the case where the execution frequency of theprocessing arranged to detect the input operation is low.

(2)

The mobile terminal device according to said (1), including:

a display unit configured to display an image,

as the controlled unit,

wherein when the input detection unit detects that the specified inputoperation is performed on the touch panel in a suspend state,

the operation determination unit causes the display unit to display aspecified screen image.

(3)

The mobile terminal device according to said (1) or (2), including:

an imaging unit having an optical system that shoots a subject, and

an imaging drive unit configured to control an operation of the imagingunit,

wherein when the input detection unit detects that an input operationdifferent from the specified input operation is performed on the touchpanel in the suspend state,

the operation determination unit starts the operation of the imagingunit through the imaging drive unit, and causes the display unit todisplay an image obtained by imaging performed with the imaging unit.

(4)

The mobile terminal device according to any one of said (1) to (3),including:

a sound emitting unit configured to emit a voice,

as the controlled unit, and

a voice processing unit configured to perform voice processing for voicedata and cause the sound emitting unit to emit a voice,

wherein when the input detection unit detects that a specified inputoperation is performed on the touch panel in the suspend state,

the operation determination unit causes the voice processing to startperforming voice processing and causes the sound emitting unit to emit avoice.

(5)

The mobile terminal device according to any one of said (1) to (4),further including:

a sensor unit configured to output a specified sensor value,

wherein when the input detection unit detects that a specified inputoperation is performed on the touch panel in the suspend state,

the operation determination unit selects a single the controlled unitfrom among plural types of the controlled units based on the sensorvalue, and causes the selected controlled unit to execute a specifiedoperation.

(6)

The mobile terminal device according to any one of said (1) to (5),

wherein when the input detection unit detects that different inputoperations are performed on the touch panel plural times in apredetermined order in the suspend state,

the operation determination unit causes the controlled unit associatedwith a different input operation to execute a specified operation.

(7)

The mobile terminal device according to any one of said (1) to (6),

wherein when the input detection unit detects that the specified inputoperation is similar to a stipulated input operation, the operationdetermination unit causes the display unit to display a guiderepresenting the stipulated input operation, and instructs thecontrolled unit associated with the stipulated input operation toexecute a specified operation when the input detection unit detects thatan input operation performed on the touch panel again in accordance withthe guide is the stipulated input operation.

(8)

The mobile terminal device according to any one of said (1) to (7),

wherein when the pattern of the input operation is detected in thesuspend state, the operation determination unit starts middleware, andcauses a specified operation to be executed from the middleware for thecontrolled unit determined to be an operation target in relation to thepattern of the input operation.

(9)

The mobile terminal device according to any one of said (1) to (8),

wherein the suspend state is a state where at least application softwareand the middleware are not started, and

wherein, in the controlled unit, an operation is controlled via theapplication software and the middleware when the input operationperformed on the touch panel is achieved in a state where theapplication software is started, and an operation is controlled via themiddleware in the suspend state.

(10)

An operation method including:

a step of detecting a pattern of an input operation based on a positionof an indicator by which an input operation is performed on a touchpanel,

a step of detecting the pattern of the input operation based on theposition of the indicator on the touch panel, and decreasing a frequencyof execution of processing arranged to detect the input operationperformed on the touch panel when the input operation performed on thetouch panel is not detected over a specified time period, and a step ofinstructing a controlled unit determined to be an operation target basedon the pattern of the input operation to execute a specified operationwhen the pattern of the input operation is detected in the case wherethe execution frequency of the processing arranged to detect the inputoperation is low.

(11)

A program causing a computer to execute:

a procedure for detecting a pattern of an input operation based on aposition of an indicator by which an input operation is performed on atouch panel,

a procedure for detecting the pattern of the input operation based onthe position of the indicator on the touch panel, and decreasing afrequency of execution of processing arranged to detect the inputoperation performed on the touch panel when the input operationperformed on the touch panel is not detected over a specified timeperiod, and

a procedure for instructing a controlled unit determined to be anoperation target based on the pattern of the input operation to executea specified operation when the pattern of the input operation isdetected in the case where the execution frequency of the processingarranged to detect the input operation is low.

(12)

A computer-readable storage medium storing a program causing a computerto execute:

a procedure for detecting a pattern of an input operation based on aposition of an indicator by which an input operation is performed on atouch panel,

a procedure for detecting the pattern of the input operation based onthe position of the indicator on the touch panel, and decreasing afrequency of execution of processing arranged to detect the inputoperation performed on the touch panel when the input operationperformed on the touch panel is not detected over a specified timeperiod, and

a procedure for instructing a controlled unit determined to be anoperation target based on the pattern of the input operation to executea specified operation when the pattern of the input operation isdetected in the case where the execution frequency of the processingarranged to detect the input operation is low.

REFERENCE SIGNS LIST

7: touch panel, 10: mobile terminal device, 39: display panel, 71: touchsensor, 72: touch sensor drive unit, 73: display unit

1. A mobile terminal device comprising: a display; a touch panelconfigured to detect input operations; and circuitry configured to:control the terminal to enter a suspend state so that the display doesnot display an image when an input operation is not detected on thetouch panel for a predetermined time period and that the detection ratein the suspend state is lower than the detection rate in an activestate; control the display to display a home screen without a passwordinput by a user when the touch panel detects a pattern of inputoperations based on the position of an indicator touching or approachingthe touch panel during the suspend state.
 2. A mobile terminal deviceaccording to claim 1, wherein the input operation is a touch operationon the touch panel.
 3. A mobile terminal device according to claim 1,wherein the touch panel is a capacitance touch panel.
 4. A method ofoperating a mobile terminal device, the method comprising the steps of:detecting input operations on a touch panel; controlling the terminaldevice to enter a suspend state so that no image is displayed when aninput operation is not detected on the touch panel for a predeterminedtime period and that the detection rate in the suspend state is lowerthan the detection rate in an active state; displaying a home screenwithout a password input by a user when the touch panel detects apattern of input operations based on the position of an indicatortouching or approaching the touch panel during the suspend state.
 5. Amethod according to claim 4, wherein the input operation is a touchoperation on the touch panel.
 6. A method according to claim 4, whereinthe touch panel is a capacitance touch panel.
 7. A non-transitorycomputer-readable medium including computer program instructions, whichwhen executed by a mobile terminal device, cause the mobile terminaldevice to: detect input operations on a touch panel; control theterminal device to enter a suspend state so that no image is displayedwhen an input operation is not detected on the touch panel for apredetermined time period and that the detection rate in the suspendstate is lower than the detection rate in an active state; and display ahome screen without a password input by a user when the touch paneldetects a pattern of input operations based on the position of anindicator touching or approaching the touch panel during the suspendstate.